The Functional Correspondence Problem

The ability to find correspondences in visual data is the essence of most computer vision tasks. But what are the right correspondences? The task of visual correspondence is well defined for two different images of same object instance. In case of two images of objects belonging to same category, visual correspondence is reasonably well-defined in most cases. But what about correspondence between two objects of completely different category – e.g., a shoe and a bottle? Does there exist any correspondence? Inspired by humans’ ability to: (a) generalize beyond semantic categories and; (b) infer functional affordances, we introduce the problem of functional correspondences in this paper. Given images of two objects, we ask a simple question: what is the set of correspondences between these two images for a given task? For example, what are the correspondences between a bottle and shoe for the task of pounding or the task of pouring. We introduce a new dataset: FunKPoint that has ground truth correspondences for 10 tasks and 20 object categories. We also introduce a modular task-driven representation for attacking this problem and demonstrate that our learned representation is effective for this task. But most importantly, because our supervision signal is not bound by semantics, we show that our learned representation can generalize better on few-shot classification problem. We hope this paper will inspire our community to think beyond semantics and focus more on cross-category generalization and learning representations for robotics tasks.


Introduction
To perceive an affordance is not to classify an object. The fact that a stone is a missile does not imply that it cannot be other things as well. It can be a paperweight, a bookend, a hammer, or a pendulum bob.

James J. Gibson
Computer vision and visual representation learning has been bound by shackles of semantic categories. Our training data is built with semantic categories -ImageNet has * Authors contributed equally Figure 1: Given a pair of images, functional correspondence establish correspondence between points that are functionally the same. In this example, we hold the body of the bottle when pouring but the neck when pounding, therefore we can establish correspondence (bottle body, shoe front) for pouring and correspondence (bottle neck, shoe front) for pounding.
1K categories of breeds of dogs, cats and mushrooms. Our supervision is semantic categories. And our evaluation tasks are semantic -image classification, object detection, image segmentation and list goes on. So it is not surprising that our approaches are bound by the limits of semantic categories. Our representations are not effective in capturing affordances for robotics tasks. And our representations fail to generalize effectively to new object categories due to focus on learning intra-class invariances. On the other hand, humans have marvelous ability to think beyond categories. We can use a screwdriver for opening screws but also to clean printer, hammer nails and what-not. Clearly, our current semantically-driven computer vision needs rethinking.
In classical computer vision, semantics did not play such an important role. Instead, correspondence was cited as one of the most important tasks in the field of computer vision. It is also the fundamental goal of visual representation learning -an embedding space where similar objects/parts/pixels have similar embedding. In an anecdotal conversation about the three most important problems in computer vision, Takeo Kanade stated that they are "Correspondence, Correspondence, Correspondence". Yet, this fundamental task of visual correspondence is ambiguous and ill-defined. What is visual correspondence? Does there exist correspondence between any pair of images? The visual correspondence problem is most well-defined and often studied in context of tracking and multi-view reconstruction where the goal is to create correspondences between two images of same object [55]. It has also been studied in the context of semantic categories where the goal is to create correspondences between images of object instances from same categories [29,39]. But it often stops at cat what are the right correspondences between two seemingly different object categories (for example, a bottle and a shoe)?
In contrast, we humans can identify correspondences between semantically different objects. We unconsciously use this ability to transfer our object manipulation skills to novel objects in order to efficiently accomplish everyday tasks. Specifically, humans possess three interesting capabilities: (a) the ability to visually infer affordances for objects, (b) the ability to generalize beyond semantic categories and (c) the ability to adapt affordances for different tasks. In order to facilitate exploration of these capabilities, we introduce the problem of functional correspondence. Given images of two objects, we ask a simple question: for a given task, what would be the set of correspondence between two objects? For example, the correspondences between shoe and bottle for the task of pouring are shown in the figure 1. The grasp locations are shown by green, storage by orange and pouring spout by red keypoints. On the the other hand, the correspondences between shoe and bottle for the task of pounding (hitting with a force) are quite different and shown in figure 1. Note that the correspondence between two objects is driven by both 3D shape and physical/material properties.
We also introduce a new dataset called FunKPoint (Section 3). FunKPoint has ground-truth keypoints labeled for 10 tasks across 20 object categories. We also propose a modular task-driven architecture. More specifically, our modular architecture computes the image representation given an input task. We show our architecture is highly effective in modeling functional correspondences although there is still a significant gap with respect to human performance. But most importantly, in proof-of-concept experiments, we demonstrate the underlying promise of learning functional correspondence. Because our task has functional supervision and there is cross-category supervision, our representation can outperform semantically-learned representations for fewshot learning.

Why Functional Correspondence?
In this paper, we introduce the problem of functional correspondence. We believe this task forms the core of visual learning because of the following reasons: (a) Object Affordances and Functional Representations: Ability to predict object affordances is a cornerstone of human intelligence and a key requirement for robotics tasks. The task of functional correspondence allows us to learn functional representations useful for robotics tasks. But more importantly, beyond predicting primary affordances (screwdriver is used for screwing), humans are really good at predicting secondary affordances (how we can use novel objects to fulfil the task -e.g. using screwdriver to clean paper jam in printer). Modeling functional correspondences across different object categories should help in predicting novel use of objects.
(b) Generalization Beyond Semantic Categories: Unlike other vision tasks such as object classification/detection or even learning 3D from image collections, this task cuts across object semantics. It attempts to model commonalities across different categories of object and hence open up the possibility of generalization beyond semantic categories.
(c) Task-Driven Representation: Finally, the groundtruth is conditioned on the task itself, the correspondences between pair of objects depends on how you envision using these objects. This allows us to formulate a task-driven representation (unlike current existing task-agnostic ConvNet representations).

Related Work
Correspondences: The correspondence problem has always been a focus of the computer vision community, and many sub-problems have been proposed with solutions offered. The classical correspondence problem establish correspondence between different views of the same object. Such correspondence is crucial for multi view geometry based algorithms and are typically solved by matching local descriptors of interesting points [3,4,21,35,36]. More recently, researchers looked into category-level correspondence [46,47,52,29,39], which does not restrict correspondence to a single instance. Such methods often model correspondence in deep feature space, and relies on simulated transformations for training. Because object of the same category usually perform similar actions, our work could also establish correspondence at category level. However, we consider any object, regardless of its object class, could correspond if they share parts that have similar functional semantics. Thus, our functional correspondence could be considered more general in that we also establish crosscategory correspondences.
Dense correspondence between pixels across video frames (optical flow) is also studied as a separate problem. Traditionally, the optical flow estimation problem is addressed as an energy minimization problem based on color constancy [23,5,44,50]. Recent optical flow estimation algorithms make use of neural networks [24,1,27,28] as models and explores self-supervision as the training method [34,17]. Another line of work focuses on the midlevel optical flow problem [55,32,26] where consistency between the regions around the pixels is also considered. Such approaches often leverage the spatial temporal coherence nature of videos to provides a natural supervision signal.
However, because the main training loss is usually a photometric loss, the learned correspondence is inevitably local. In this work, we try to establish a higher level functional correspondence. Such correspondence involves a knowledge of object affordances, which is still hard to learn from unlabeled raw videos. Functional Representations and Affordances: The core idea of affordances was introduced by James J. Gibson [16]. Gibson described object affordances as "opportunities for interactions". Inspired by Gibson's idea of affordances, a long-term goal for robotic perception has been to perform function recognition [51,45]. Approaches such as [51,57] used manually-defined rules to predict affordances. However, these approaches were too brittle and failed to generalize.
In recent years, with the advances in 3D scene understanding and with the large-scale availability of interaction data the idea of affordances has been revisited as well [20,18,15,59,7]. Approaches such as [20,59] have attempted to use 3D understanding followed by affordance estimation. More recently, approaches have tried to collect large-scale data for affordance estimation [54] and used ConvNets to predict affordances in the scene [14,53]. Affor-danceNet [10] simultaneously localizes multiple objects and predicts pixel-wise affordances by training on a large-scale dataset with affordance labels. In contrast, our approach focuses on affordances as a vehicle to target generalization beyond semantic categories and learn task-driven representations. More specifically, we target using primary and secondary object affordances to learn visual correspondences across different object categories. Our work is also closely related to some recent work in robotics which focuses on extraction of keypoints for robotics tasks [13,37]. However, in most of these scenarios, the goal is to learn to predict dense keypoints/correspondences across two objects of same categories. In this work, we focus on the more general problem of how to do task-driven functional correspondences across multiple object categories. Task-Driven Representations and Modular Networks: Classification models in deep learning have largely been trained as discriminative models [30,49,22]. Recently, energy based models [33] have gained popularity and demonstrated success on image classification [19], continual learning [12], compositional zero-shot learning [42,56] and generative modeling of text [2]. In [42,56], the key idea is to construct a task-dependent (or label-dependent) neural network for classifying whether an image belongs to the considered label. In [42], this compatibility of an image x to a label y is computed using a sequence of neural network modules which are reweighted using a function of the considered label y. The modular architecture proposed in [42] allows sharing of learned filters across different labels which is crucial for domains where the labels are heavily related. These modular neural networks have also demonstrated great success in multi-task reinforcement learning [9,58] where modules are shared among related tasks to learn policies efficiently. For estimating functional correspondences, we require representations that vary according to the considered task. Therefore, we adopt a similar modular task-driven architecture for learning a task-dependent representation which also allows us to share neural network modules between related tasks.

The FunKPoint Dataset
To explore the study of functional correspondences, we present a novel dataset: FunKPoint (short for Functional KeyPoints). FunKPoint consists of 2K objects covering 20 object categories. In order to learn and evaluate functional correspondences between pairs of images, we require dense human annotations of such correspondences. However, such an approach is unscalable due to the quadratic number of image pairs and pixels. Instead, we first identify 5 semantically meaningful points that are essential for each task. For each task, we then collect annotations for the 5 keypoints for each relevant object image. Figure 2 shows examples from the dataset. Note that a single image could be labeled differently for each task. In total, around 24K such labeled keypoints are obtained. Any two objects that can be used to perform an action are then used to establish a correspondence relationship (w.r.t. that action). This correspondence between two images, conditioned on a specific action, is referred to as a Functional Correspondence. For example, in the top left figure of Fig. 2, both hammers can be used to pull out a nail, so a functional correspondence relationship (consists of 5 pairs of corresponding points) could be established between the two objects. Similarly, both the spoon and the frying pan ( Fig. 2 top-middle) can be used to scoop things, so we can also generate a functional correspondence relationship between them.
Data Collection First, we curate an action vocabulary consisting 10 common tasks (or actions). Our action vocabulary is inspired from the TaskGrasp [38] dataset, which focuses on task-dependent robot grasps. Therefore, the 10 actions in our vocabulary are not only common, but also useful as a benchmark in robotics. For each action, we identify 5 object categories that can be used to perform that task. Note that many object categories can be relevant for multiple tasks. This allows us to generate different correspondence for the same objects under the condition of performing different tasks. For example, the object category frying pan has 2 possible actions (among others): Scoop and Mash/Pound. The rim of the pan is a functional keypoint that is important for scooping, but for pounding, the bottom of the pan becomes the relevant functional keypoint. See Table 1 for the list of 20 objects and their associated tasks.
For each of the 20 object categories, we collect 100 im-  Associating keypoints between images provides us with numerous intra-category and inter-category functional correspondences. ages from the ImageNet dataset [8], but supplementing with creative commons images from Google image search to reach 100. Note we manually filter out images that contain multiple object instances, missing parts or occluded parts. We use Amazon Mechanical Turk to collect human annotations for the keypoints. Each (image, task) pair is labeled with the 5 functional key points as well as a choice of labelling difficulty (between easy, medium or hard). In the interface, we provide a simple definition for each point, current action, and also examples of labeled images. See supplementary material for a visualization of the interface. As explained, each object could be associated with multiple actions (see Supp. for statistics). From the collected data, we create a train split containing 4044 (image, task) pairs and a test split contains 741 (image, task) pairs.

Approach
Estimating semantic correspondences has been well studied in the past. Most approaches [46,47,52,29,39] involve learning a pixel or patch level representation which can be used to match corresponding points on similar objects. As we will demonstrate via experiments, for the problem of functional correspondence, such representations are not suitable. We wish to estimate correspondences even across semantically varied objects and second, the correspondences vary according to the task being performed. Therefore, we propose an approach that produces task-driven representations that can be used to find functional correspondences across varied objects.
First, we formalize the problem setup of functional correspondence. Consider two images I depicting an object o and I depicting an object o such that both objects can be used to perform task t. Given any point p on object o, the goal of the functional correspondence problem is to estimate the functionally corresponding location p on object o . However, as described in Sec 3, we only have access to correspondences for specific keypoints due to the prohibitive cost of annotation. Therefore, for each task t ∈ T that can be performed with object o, we have a set of functional keypoints {p t1 , p t2 , ..., p tK }. The goal of the functional correspondence problem can then be restated as estimation of the functionally corresponding locations of the keypoints {p t1 , p t2 , ..., p tK } on object o .
Recently, task-driven classifiers have gained popularity for the problem of zero-shot learning [42,56]. Taking inspiration from these approaches, we adopt a similar approach to learn a task-driven representation. More formally, we propose a model F θ with parameters θ which takes as input an image I, a task t and outputs a representation f = F θ (I, t).
In order for the representation f to be useful for functional correspondence, we propose to learn the parameters θ using the dataset presented in Sec 3. The goal is to ensure that the representation f at location p of an image I and location p of an image I are identical only when p, p are functional correspondences. To achieve this, we propose a contrastive learning objective [40] as follows: is the indexed feature f at spatial location p here p tk , p tk are the k-th functional keypoints for task t in image images I, I respectively. Intuitively, minimizing this objective effectively minimizes the distance between features of functionally corresponding points in the two images (numerator) and maximizes the distance between the feature of a keypoint and the features at all non-corresponding locations p (denominator). Note that the locations p includes all keypoint and non-keypoint locations. This general contrastive learning formulation can be applied to any convolutional neural network architecture that jointly encodes the image I and task t. In order to model the dependencies between functional keypoints of different tasks, we propose to use a modular architecture allowing us to share filters across tasks. We adopt the architecture proposed in [42]. For the sake of completeness, we describe the architecture here in detail. Figure 3 shows an overview of our proposed model F. For an image I, we first extract task-agnostic features using a ResNet trunk upto the conv4_x layer (defined in [22]) as r = R(I). For an image with dimensions H × W , the representation r has spatial dimensions H/16×W/16 with a C dimensional feature at each location. The representation r is then processed by a modular task-driven feature extractor T to produce the final features f = T (I, t).

Implementation Details
The modular task-driven feature extractor T consists of N T layers with each layer comprising of M T modules ex-cept for the last layer which comprises of a single module. A module can be any differentiable operation. In our proposed architecture, we use convolution layers with batch normalization [25] and ReLU activation functions (see supplementary for details of kernel size, number of filters, etc). We denote the j-th module of the i-th layer as T ij . Given a task-dependent weight tensor W t ∈ R N T −1×M T ×M T for task t, the output of a module T ij is computed as: Intuitively, the input to a module is a weighted sum of the outputs of the modules in the previous layer. For modules in the first layer, the inputs are taken as the task agnostic representation produced previously i.e. o 0k = r. Finally, the output task-dependent representation is taken as the input of last module o (N T )1 . Note that we assumed that we are given a task-dependent weight tensor W t ∈ R N T −1×M T ×M T . This weight tensor is estimated using a separate fully-connected neural network G known as the gating network (see supplementary for parameter details). The gating network takes as input a task-embedding t and outputs the weight tensor as W t = G(t). As explained in [42], the input weights to each module (W [i, j, :]) needs to be projected to the probability simplex using a softmax operation to encourage separate paths for different tasks. In summary, the output feature representation is computed as f = T R(I), G(t) .

Training
The objective presented in Equation 1 is used to learn the parameters θ which comprises of the gating network G, modules T ij and task embeddings t (which are initialized randomly for each task). We pretrain the ResNet model R on ImageNet [8] and fix its parameters. We optimize the parameters using SGD with a learning rate of 0.01, weight decay of 0.00001 and momentum of 0.9. Each batch consists of 256 pairs of images randomly sampled from the training split of the FunKPoint dataset.

Experiments
Modeling functional correspondences provides numerous practical benefits. In this section, we demonstrate this by evaluating our presented model on a suite of tasks. First, we show that our model can effectively identify functional correspondences and outperform numerous baseline methods. We then demonstrate the efficacy of our learned representation for few-shot learning, grasp prediction, and ADROIT manipulation tasks [43]. Note that due to the domain of our training data, we focus our experiments on manipulation related datasets for all tasks.

Functional Correspondences
We first evaluate the performance of our model on the task of estimating functional correspondences. We create an evaluation benchmark of (training image,test image, task) triplets using the FunKPointdataset. As explained earlier, each image is associated with 5 keypoint annotations. The goal is to identify the location of each keypoint in the test image using the associated train image and task.
Given representations of the train and test images f train , f test , the corresponding test image location for a training image keypoint p train can be identified as: We use the PCK metric to evaluate the quality of estimated keypoints. An estimated keypoint in the test image is considered correct if it lies within 23 pixels from the ground truth annotation.
The ImageNet baseline involves using features from a pretrained ResNet-50 conv4_x layer (same as the trunk of our model) to compute the correspondences. We also compare to self-supervised semantic correspondence estimation model presented in [31] and features from a task-agnostic affordance estimation method [11]. Finally, we evaluate three variants of our model. Ours with task-embedding refers to our full model. Ours without task-embedding (uniform) refers to a model with W t [i, j, k] = 1/M T i.e. the gating weights are constant, uniform and not dependent on the task embedding t. Ours without task-embedding (learned) refers to a similar task-independent model, where a single learned gating weight W t is shared for all tasks. We observe that our proposed model substantially outperforms the ImageNet model and self-supervised learning methods. This further illustrates the difference between learning representations for semantic and functional correspondence. In the ablation of our model, we observe that our proposed model outperforms its task-independent variants by a substantial margin. This emphasizes the need for task-dependent features since functional correspondences are closely tied to the task considered.
We also train a variant of our task-dependent model by initializing from a self-supervised learning method DINO [6]. We observe that while this underperforms the model initialized from ImageNet, it still significantly outperforms all the baseline methods. This indicates that learning using the FunKPoint dataset is crucial.
Finally, we measure the consistency of annotations across humans in the functional correspondences by collecting a second set of human annotations. We observe that the new annotations of correspondence achieve a PCK of 82.5%. Additionally, on a subset of randomly chosen 200 pairs of images, we collected annotations from 4 humans. We observe that the median distance between estimated functional keypoints was 13.07 pixels. These results demonstrate the ambiguity in the functional correspondence task is minimal.
In Figure 4, we present a visualization of the estimated correspondences for five tasks. We observe that our model is able to learn inter-category correspondences. For example, it is able to learn correspondence between bottlehead and pan spouts for pouring. Some interesting correspondences include correspondence between hammerhead and sole of the shoe and correspondence between spout and tip of gardening-tool. While our model was trained to estimate correspondences for keypoints, our model learns to estimate correspondences for all points on objects. In Figure 5, we visualize densely sampled points on objects and their estimated correspondences on test images. While our model is trained on 5 key points in each image, we observe that the model can approximately associate each densely sampled locations on the reference object to the functionally appropriate location on the target object. For example, the rim of the mug in the first image is appropriately associated with the spout of the watering can.

Few-shot Generalization
Classification of objects requires understanding its appearance and 3D structure. However, exhaustively modeling appearance and 3D properties from a few samples is challenging and ambiguous in many cases. For example, observing an image of a white conical coffee mug could lead to the belief that all mugs are conical. What we need is a way to use the data from other categories to help learn what makes mug a mug? Since, in the task of functional correspondence, we already label correspondences across multiple categories, our learned model might have better ability to create cross-category generalization. This is the hypothesis we want to test in this experiment.
First, we curate a small dataset of 5 manipulable objects (shovel, water jug, coffee mug, wok and letter opener) with 20 images each. We create train-test splits by including 1,  Figure 4: Qualitative Correspondences. We demonstrate some qualitative correspondences generated by our algorithm. Squares indicate predicted functionally corresponding keypoints in the second image and circles indicate ground truth keypoints in both images. Notice how the correspondences differ for input task. For example, for the task of mashing/pounding the bottle base corresponds with hammerhead. Similarly for poking, the spout of watering can corresponds to the tip of gardening tool.  2 or 5 images for each object in the train set and the rest in the test set. In each of the settings, we generate 3 different random samples for the splits leading to a total of 9 unique splits. We train a linear classifier to classify the features extracted from our proposed model. Since our model extracts task-dependent features, for each image, we concatenate the features extracted for all 10-tasks and perform spatial average pooling to reduce the dimensionality. As a baseline, we similarly train a linear classifier on the ResNet-50 conv4_x features pretrained on ImageNet. For a fair comparison, we also finetune an ImageNet representation to classify objects in our presented FunKPoint dataset. We present the results in Table 3. The representation learned by our model outperforms the ImageNet based representation on all three settings by substantial margins. We also note that our model outperforms the representation optimized for classifying the objects in the FunKPoint dataset. This indicates that the task of functional correspondence leads to representations that generalize better to novel manipulable objects.

Grasp Prediction
Functional representations are ideally suited for facilitating downstream robotic manipulation tasks. A common challenge addressed in robotic manipulation is the task of grasp prediction. In [41], this is formalized as prediction of grasp success given an image and a hypothesized grasp angle. For this task, we evaluate the efficacy of the features learned by performing functional correspondence. We extract features using our proposed model, concatenate with the hypothesized grasp angle (as a discretized 18-way onehot vector), the extracted feature is then fed into a 2-layer neural network appended at the end of the modular network to predict the grasp success label. The feature extractor and the classifier are jointly finetuned on the training set of the benchmark using a smaller learning rate of 2e −4 , until the model converges. As a baseline, we use an ImageNet-based ResNet-50 (similarly truncated at layer3 as ours). Table 4 contains the numerical results for our model on the Grasp benchmark. Our method outperform the baseline method by 1.7% accuracy, demonstrating the advantage of functional representations.

ADROIT Manipulation Task
A lot of recent research has focused on learning robotic manipulation of objects through reinforcement learning. In this section, we investigate whether standard reinforcement learning (RL) based methods can take advantage of functional representations. We adopt the method proposed in RRL [48], a simple RL algorithm that uses pre-trained ResNet [22] features which can be easily replaced by our representation. We evaluate this algorithm on the ADROIT manipulation suite [43], which consists of several complex dexterous manipulation tasks. In the Tool Use task environment, we evaluate for the task of hammering a nail.
In Figure 6, we present the success rate of our representation compared to the baseline ImageNet-based based features. We observe that our representation leads to improved sample efficiency and final performance at convergence. We believe these results demonstrate the promise of functional representations for robotics problems. We hope that this will  . We observe that our functional representation demonstrates improved sample efficiency and success rate compared to a baseline ImageNet-based representations that does not encode functional aspects of objects.
inspire more exhaustive investigations of functional representations and their role in robotics.

A. Appendix
A.1. Annotation Interface 3. The worker must add annotation for the difficulty.
The labelling process took around 5 days. We then check for errors in the annotations and relabel as described in the main text. Figure 7 shows the level of difficulties provided by the annotators. Note annotators could be different for different categories, and the difficulty values may not be consistent across all annotators (different annotators may feel different difficulty for labelling the same image). Here, 0 means Easy, 0.5 means Medium and 1 means Hard. The values shown are computed from an average over all objects in the class. As one could expect, screwdriver is the easiest object category because there are very little ambiguities in defintions of each point and the shape variations are small. Two most difficult object classes are baskets and dustpan. The potential reason could be their large shape variance. For baskets, there are woven basket, shopping basket, basket with lids, without lids, and many others. Similarly, dustpan could have different handle length and orientation. Some so called lobby dustpans could have another structure that functions as a lid. As comparison, the easier object classes such as bottle, cup and tablefork have relatively little shape variance.

A.3. Implementation Details
In this section, we provide additional hyperparameter details to facilitate easy reproduction of our results.
As explained in Sec 4.1 of the main text, our proposed model is inspired from the task-driven modular networks proposed in [42]. We design the modular network as 4 layers with 6,6,6,1 modules in each layer respectively. Here we present the hyperparameters of the convolution layers: 2nd layer: 128 filters, kernel size=3, stride=1, padding=1 3rd layer: 128 filters, kernel size=3, stride=1, padding=1 4th layer: 128 filters, kernel size=1, stride=1, padding=0 We train the modules using SGD with learning rate 0.01, momentum 0.9 and weight decay 0.00001 with a batch size of 256. The gating network takes as input a 100-dimensional embedding based on the task under consideration. The 10 embeddings for the 10 tasks in the FunKPoint dataset are randomly initialize and learned during the optimization process. The gating network consists of a 2-layer fully-connected neural network with a hidden embedding size of 100.

B. Dataset Statistics
As explained in the main text, each object could be associated with multiple actions. This leads to varying number of keypoint annotations based on object category. In Figure  9, we present these statistics :   scrub brush  toothbrush  dustpan  scissors  screwdriver  tongs  trowel  whisk  watering_can  basket  bottle  hammer  shoe  cup  scraper  spatula  frying_pan  ladle  tablefork  spoon  0  500  1000  1500  2000 2500 3000